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ABSTRACT 

An ASEA IRB90 robotic 
manipulator with attached inspection 
cameras was moved through a Space 
Shuttle Payload Assist Module (PAM) 
Cradle under computer control. The 
Operator and Operator Control 
Station, including graphics 
simulation, gross-motion spatial 
planning, and machine vision 
processing, were located at the Jet 
Propulsion Laboratory (JPL) in 
California. The Safety and Support 
personnel, PAM Cradle, IRB90, and 
image acquisition system, were 
stationed at the Kennedy Space 
Center {KSC) in Florida. Images 
captured at KSC were used both for 
processing by a machine vision 
system at JPL, and for inspection by 
the JPL Operator. The system found 
collision-free paths through the PAM 
Cradle, demonstrated accurate 
knowledge of the location of both 
objects of interest and obstacles, 
and operated with a communication 
delay of two seconds. Safe operation 
of the IRB90 near Shuttle flight 
hardware was obtained both through 
the use of a gross-motion spatial 
planner developed at JPL using 
artificial intelligence techniques, 
and infra-red beams and pressure 
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Jet Propulsion Laboratory, 4800 Oak 
Grove Drive, Pasadena, California 
91109, Mail Stop 301-250D or at 
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sensitive strips mounted to the 
critical surfaces of the flight 
hardware at KSC. The Demonstration 
showed that telerobotics is 
effective for real tasks, safe for 
personnel and hardware, and highly 
productive and reliable for Shuttle 
payload operations and Space Station 
external operations. 

BACKGROUND* 

Telerobotic systems are 
typically demonstrated with the 
operator in close proximity to the 
robot and with nearly instantaneous 
feedback to direct subsequent 
actions . However, many applications 
require ground-based control of 
remote space-based robots or local 
control over low-data-rate networks, 
each of which introduces a 
significant communication time delay 
that alters the nature of the 
operator interaction. Proposed 
solutions to the time delay problem, 
including remote site autonomy and a 
high-level operator interface, need 
to be tested in an environment where 
the delays are present . 

Inspection tasks are typical 
of those that will be required of 
remote robots. One application of 
telerobotics is for Space Shuttle 
payload processing. To inspect 
Shuttle payloads, technicians walk 


* Adapted from Jet Propulsion 
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above flight hardware to obtain 
access, and must rely on safety 
harnesses and expensive temporary 
scaffolding. A telerobotic system 
could significantly reduce the cost 
of payload inspection, and greatly 
improve the safety of the personnel, 
the payload, and the Shuttle. 

While communication time delay 
can be avoided in an operational 
Shuttle "payload inspection robot," 
autonomous operation and high-level 
control would improve the cost- 
effectiveness and safety of the 
system. By building and 
demonstrating a prototype that can 
be controlled either locally or from 
a remote site, progress in both 
Shuttle operations and space tele- 
robotics can be achieved. 

SPACE FLIGHT PROBLEM DOMAIN 
Space Shuttle Payload Operations * 

At KSC, access to payloads 
during pre-launch payload operations 
is very restricted. At the Operation 
and Checkout Building, where 
horizontal payloads are integrated 
into the payload bay, work-stands 
are sometimes built to lower 
technicians down between satellites 
to retrieve, replace or connect an 
object. After the integration of the 
horizontal payloads, the Shuttle is 
mated to its solid rocket boosters 
and external tank, and rolled out to 
the launch pad. Payloads which have 
to be integrated into the payload 
bay in a vertical configuration are 
first inserted into a canister at 
the Vertical Processing Facility, 
and are then shipped to the launch 
pad for integration. When the 
canister arrives at the launch pad, 
it is lifted into the Payload 


* For a thorough discussion of Space 
Shuttle Payload Operations see 
Kennedy Space Center [KSC], 1978. 


Changeout Room (PCR) . The PCR is a 
clean-room integrated into the 
Rotating Service Structure (RSS) ; 
the RSS is rotated against the 
Shuttle during pre-launch servicing 
activities . 

The payload is first removed 
from the cannister and brought 
inside the PCR by the Payload Ground 
Handling Mechanism { PGHM) . The PGHM 
is a very large device on an 
overhead beam that removes the 
payload from the cannister and 
inserts it into the payload bay. The 
RSS is then rotated into place in 
front of the payload bay, and the 
payload is moved into place. 

When the payload has been 
inserted into the payload bay, it is 
not visible beyond the PGHM. Limited 
access to the payload is possible by 
crawling out onto platforms. "C" 
clamps, gangplanks and roll-out 
platforms are used to gain access 
inside the payload bay. It is 
sometimes necessary for a technician 
to climb out onto a gangplank in 
order to take close-up photographs 
or to remove lens dust-covers. A 
technician also has to remove tagged 
items just prior to launch. Twice 
for each launch, at the start of PCR 
operations and at their conclusion, 
technicians have to reach hazardous 
positions 65 feet above multi- 
million dollar payloads to attach 
grounding straps . This involves 
bolts and test gear which, if 
dropped, may cause extensive and 
costly damage to a payload, 
requiring removal and repair of the 
payload, with large "return from 
pad" consequences . 

SPACE FLIGHT OBJECTIVES 

One of the objectives of the 
JPL/KSC Inspection Demonstration was 
to aid Space Flight operations by 
demonstrating effective man/machine 
teamwork on a task that has 
applications to operational Shuttle 
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payload processing. To this end, it 
was necessary to demonstrate that a 
telerobotic system can: (a) operate 
in the complex environment of a 
Shuttle payload bay or PCR; <b) 
operate without significant risk to 
personnel, equipment or payload, 
reducing both the need for risky 
gangplank operations, and the chance 
of errors; (c) improve the 
productivity of payload operations, 
easing access to hard-to-reach 
areas; and (d) improve the 
reliability of payload operations. 

SPACE STATION PROBLEM DOMAINS 

Space Station operations for 
construction and maintenance require 
extensive access to the external 
portions of the Space Station. A 
variety of technologies exist which 
meet the need for external access, 
including Extravehicular activity 
(EVA) , Flight Telerobotic Servicer 
(FTS ) teleoperation, and ground- 
remote telerobotics, each with some 
advantages and disadvantages. 

Extravehicular Activity 

The use of EVA involves 
astronauts in space-suits performing 
assembly and servicing tasks outside 
of the Space Station. 

Advantages . 

One advantage of EVA for on- 
orbit construction and maintenance 
of the Space Station is that the 
astronauts at the work-site can 
better perceive problems and their 
solutions . 

Disadvantages . 

1. There are many risks to the 
astronaut performing EVA, including 
the possibility of death during 
Space Station construction and 
operations . 


2. Astronaut productivity is 
lower due to the difficulty of 
performing dexterous operations in a 
bulky space suit which limits touch 
and vision. 

3. There are large amounts of 
expensive astronaut on-orbit time 
required for EVA tasks, e.g. the 
required three hour pre-breathing 
period before exiting the vehicle. 

4. Limited dexterity increases 
the possibility of mistakes and 
reduces reliability and safety. 

FTS Teleoperation 

The FTS allows astronauts 
inside the Space Station to perform 
teleoperation since teleoperation 
from Earth is not practical due to 
the communication delay. 

Advantages . 

The teleoperation of the FTS 
within the shirt-sleeve environment 
of the Space Station eliminates the 
risk to the astronaut due to EVA. 

Disadvantages . 

1. The limitations of tele- 
operation contribute to low 
astronaut productivity, although 
there is a significant potential for 
improvement though telepresence. 

2. The teleoperation of the 
FTS, like EVA, requires large 
amounts of expensive astronaut on- 
orbit time. 

3. The limited dexterity 
available with teleoperation, and 
the potential mistakes, reduce 
reliability and safety. 

Ground/Remote Telerobotics 

The use of ground/remote tele- 
robotics allows operators at a 
ground-based control station to 
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operate semi-autonomous telerobot (s) 
at the Space Station. 

Advantages . 

1. Eliminates the risk due to 

EVA. 

2. Enhances productivity by 
allowing telerobotic operations to 
proceed continuously as long as 
there is work to be done, with no 
work stoppages for crew sleep or 
delays for pre-breathing. Partial 
autonomy allows one operator to 
control two or more telerobots . 

3. Minimizes astronaut on- 
orbit time required for external 
servicing tasks; all robotic control 
is performed by ground technicians. 

4. Enhances reliability since 
a telerobot can do repetitive 
assembly tasks automatically without 
boredom or distraction. 

Disadvantages . 

The operator's remoteness from 
the work-site limits the ability to 
perceive the work-site, thus making 
problem-solving more difficult and 
forcing increased reliance on 
machine autonomy. 

Actual Space Station opera- 
tions will, most likely, include 
some mixture of EVA, FTS tele- 
operation, and ground/remote tele- 
robotics, depending on requirements 
and available capabilities. 

SPACE STATION OBJECTIVES 

The task of the JPL/KSC Tele- 
robotic Demonstration was to aid 
Space Station operations by 
demonstrating effective remote task 
execution with a limited band-width, 
uncertain time delay between the 
operator control station and the 
work-site, thus overcoming the tele- 
operation time-delay problem. To 


this end, it was necessary to 
demonstrate that a telerobotic 
system can: (a) operate when the 
sensor and actuator systems are 
remote from the operator control 
station, when the communication 
band-width is limited, and when 
there is a variable communication 
delay of several seconds; (b) 
operate in a realistically complex 
flight hardware environment; (c) 
operate without significant risk to 
personnel, equipment, or payload, 
reducing both the need for EVA, and 
the probability of errors; (d) 
improve the productivity of 
operations in space by reducing the 
need for EVA thus freeing valuable 
astronaut time for other activities, 
by operating from the ground thus 
utilizing far less expensive ground- 
based personnel, and by allowing 
more time (even continuous) on- 
station; and (e) improve the 
reliability of space operations by 
reducing mistakes which might be 
made during EVA due to boredom and 
fatigue . 

THE JPL/KSC TELEROBOTIC INSPECTION 
SYSTEM 

The Robotics Applications 
Development Laboratory (RADL) at KSC 
includes a large ASEA IRB90 robotic 
manipulator on a track and various 
support computers for controlling 
the IRB90 and processing video data 
for machine vision applications. The 
ASEA IRB90 is an industrial 
ma t e r i a 1 s -handl ing robot with a 
payload capacity of approximately 
200 pounds, and a height of 
approximately nine feet . The IRB90 
has been outfitted with a dual- 
camera platform. The work-site 
includes an inert PAM and support 
cradle in a ground support equipment 
(GSE) frame. The PAM, Cradle, and 
GSE frame were all obtained from the 
manufacturer; the Cradle had flown 
on a previous Shuttle mission, and 
was to be maintained in a flight- 
ready condition . 
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The Task Planning and 
Reasoning ( TPR) and Sensing and 
Perception (S&P) subsystems were 
located at JPL, while the Arm Device 
Control (ADC) and Video Device 
Control (VDC) subsystems were at KSC 
(see Figure 1) . TPR (Peters, 
Collins , Mittman, O'Meara, and 
Rokey, in press) was implemented in 
LISP on a Symbolics LISP machine, 
and used a VAX 11/750 as a network 
communications gateway. S&P 
(Gennery, Litwin, Wilcox, and Bon, 

1987) was implemented in Pascal on a 
VAX 11/750 with 240 by 320 pixel 
frame buffers. ADC and VDC were 
implemented on a MicroVAX, with 
serial communications to the VME- 
based processors which contained the 
direct hardware interfaces to the 
IRB90 and the video cameras. 

Communication between sub- 
systems took place over DECnet using 
an application layer called the 
Network Interface Package (NIP) . The 
work-site, with IRB90, controller, 
video cameras and frame buffers, was 
located at KSC in Florida. The 
operator site, with computer and 
software providing a graphics 
operator interface, gross-motion 
spatial planning and machine vision, 
was located at JPL in California. 
Communication between the two sites 
was over a 9600 baud serial link on 
a shared network (PSCN) , resulting 
in variable and unpredictable 
communication delays which average 
two seconds per round-trip 
transaction . 

The intelligent technology 
used in the JPL software was 
primarily transferred from JPL's 
Telerobot Testbed project. This 
includes the Network Interface 
Package (NIP) used for all inter- 
subsystem communications, the 
graphical user interface (Mittman, 

1988) and gross-motion spatial 
planner (Collins & Rokey, 1988) used 
by the TPR subsystem, and the 
machine vision system used by the 


S&P subsystem. All software except 
the NIP required modifications and 
new interfaces for this task. 

Work-space models for spatial 
planning, machine vision, and the 
user interface were derived from a 
CAD database supplied by KSC. Off- 
line software utilities at JPL 
provided transforms to move all 
models into the same coordinate 
system and allow calibration of the 
cameras which supply the images for 
machine vision. The control station 
(TPR) commanded S&P to perform its 
vision functions and also commanded 
the ADC to carry out the desired 
robot motions. The S&P subsystem at 
JPL commanded stereo images to be 
transmitted from the VDC subsystem 
at KSC. Using KSC-supplied 
descriptions of camera viewpoint 
locations, the S&P subsystem 
verified the spatial object database 
required by the high-level spatial 
planner, thus ensuring the safety of 
IRB90 motions. 

New work performed for this 
task included implementation of the 
ADC and VDC subsystems at KSC, 
generation of IGES models for 
objects in the work-space, 
measurement of work-space points to 
enable calibration, transformation 
of IGES model data into the IRB9Q 
coordinate frame, conversion of IGES 
models into the forms needed by the 
JPL software, generation of free- 
space maps for use in gross-motion 
spatial planning, calibration of 
video camera models for use with 
machine vision, and implementation 
of video processing software, 
including image sub- samp ling, 
compression/decompression, and low- 
level feature extraction. 

SUMMARY OF FIRST-YEAR RESULTS 

Hardware and communications 
were installed, integrated, and 
tested. The PAM Cradle and inert PAM 
were acquired and IGES models were 
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Figure 1: Logical Block Diagram 



created. Device control software was 
developed at KSC, including 
communications with JPL software. 
JPL Sensing & Perception (S&P) 
subsystem was modified for the needs 
of this task. The JPL Task Planning 
and Reasoning (TPR) subsystem was 
modified and extended for this task, 
providing gross-motion spatial 
planning, a direct interface to KSC 
for IRB90 control and a graphical 
user interface. A successful 
capability test was performed, 
including: (a) control of a robotic 
manipulator from a distance of 3000 
miles with variable time delays 
averaging two seconds, (b) motion 
into an occluded, covered region in 
a very constricted work-space, (c) 
use of a gross-motion spatial 
planner to avoid collisions, (d) use 
of machine vision to verify location 
of modeled objects, and (e) 
operation on real flight hardware. 

DIFFICULTIES OVERCOME 

As might be expected in the 
first year of a task, numerous 
difficulties and delays arose. PSCN 
mistakenly installed a synchronous 
line instead of an asynchronous 
line, and the Symbolics NIP version 
proved to be unusable due to 
compiler incompatibilities with a 
new operating system. A VAX NIP 
server with a custom interface 
between the VAX and Symbolics 
machines was created. 

The IRB90 controller was of 
limited use because the proprietary 
nature of the information contained 
within the controller made it 
impossible to obtain accurate IRB90 
kinematic parameters. An IRB90 
kinematic model was constructed from 
the IRB90 printed documentation. 

The IRB90 controller interface 
did not accommodate joint controlled 
motion, the mode used by JPL f s 
gross-motion spatial planner. 
Motions were planned in joint space. 


then passed through the forward 
kinematics of the IRB90 model to 
derive Cartesian end-effector 
positions for commanding. 

Software was implemented to 
convert IGES model data and 
transform it into the IRB90 
coordinate frame. Limitations in the 
CAD system from which the IGES data 
originated required that conversion 
software be written with operator 
interaction to aid in designating 
IGES object connectivity. Additional 
software was implemented to compute 
a homogeneous transformation between 
IGES model and IRB90 coordinate 
systems when given a set of points 
measured in both frames. 

Camera calibration within the 
S&P subsystem was conducted with a 
poor dispersion of calibration 
points. The iterative fit of the 
camera model to the measured data 
did not converge. Existing software 
was modified to allow for the manual 
editing of the initial camera model 
estimates. Editing was accomplished 
with a graphic display showing the 
measured calibration points and 
calibration images. The elimination 
of outlying calibration points and 
the selection of a good initial 
estimate allowed the camera models 
to converge. 

POSSIBLE IMPACTS ON SPACE FLIGHT AND 
SPACE STATION 

Modifications in Requirements 

In order to provide for the 
increased activity of ground/remote 
telerobotic operations, 
modifications will need to be made 
which provide the appropriate level 
of communication with Earth. 
Modifications of the FT S for 
proximity sensors, increased video 
coverage, and required local 
processing should also be made, e.g. 
for reflex actions. An Operator 
Control Station and processing 
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facilities on Earth would also be 
required as part of the Space 
Station design. To make the 
operations more amenable to robotic 
manipulation, tools and jigs should 
be designed. 

Benefits 

The benefits of ground/remote 
telerobotics for the space station 
include a 24 hour/day work cycle for 
Space Station assembly with 
alternating ground personnel 
controlling the assembly robots, and 
improved astronaut safety through 
reduced EVA. Reliability is also 
improved by eliminating repetitive, 
menial, and tiring tasks from the 
operator’s work-load. 

FUTURE PLANS 

There are many plans for 
future work, as time and budget 
allow. The following are a sample of 
the items which will be incorporated 
into the present system at a future 
time . 

Kennedy Space Center 

1. Development of requirements 
and design proximity sensors for the 
IRB90 . 

2. Design, build and integrate 
a two degree-of-f reedom articulated 
"boom" extension to the IRB90. 

3. Design, build and integrate 
a video camera system for the 
extended IRB90 . 

4. Develop an accurate 
kinematic model of the extended 
IRB90 . 

5. Install the TPR subsystem 
software on a artificial 
intelligence workstation located at 
KSC . 


Jet Propulsion Labora to ry 

1 . Expansion of the IGES world 
model to allow for more flexible 
operations, and more viewpoints. 
This requires the addition of a 
fine-motion spatial planner. 

2 . Improvement of the operator 
interface and overall system speed. 

3. Addition of fine-motion 
spatial planning to enable the IRB90 
to move to arbitrary positions. 

4. Development of models for a 
modified IRB90 and a new camera 
system. 

5. Transition of the machine 
vision system to a next-generation 
VME-based hardware platform. 

FUTURE CHALLENGES 

The JPL/KSC team faces some 
future challenges which can be met 
by a well-designed research effort. 

Proximity Sensing 

The design of the proximity 
sensors should aid in increasing 
safety, while the information from 
the sensors should be utilized for 
spatial planning. 

Spatial Planning 

1. Improvement of gross-motion 
spatial planning by speeding the 
graph generation. 

2. Integration of fine-motion 
with gross-motion spatial planning. 

3. Development of spatial 
planning for an incompletely or 
erroneously modeled environment. 

4. Integration of spatial 
planning tools with operator 
interface to resolve spatial 
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problems and to make the spatial 
planning faster and more reliable. 

Perception 

1. Localization of objects 
when a priori location is unknown. 

2. Characterization of known 
objects . 

3. Effective modeling of a 
complex environment . 
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